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METHOD AND APPARATUS FOR AUTOMATICALLY CREATING 
DATABASE FOR USE IN AUTOMATED MEDIA RECOGNITION SYSTEM 

RELATED APPLICATIONS 

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 
60/273,146, filed March 2, 2001 with the same title and by the same inventors and 
assigned to the same Assignee. 

[0002] This application is related to U.S. Patent Applications Nos. 09/671 ,57 1 
entitled "Method and System for Purchasing Pre-Recorded Music" filed on September 27, 
2000 by the same inventors; and 09/734,949 entitled "Method and System for Interacting 
with a User in an Experiential Environment" filed on December 12, 2000 by the same 
inventors. This application is also related to U.S. Provisional Patent Application No. 
60/222,023 entitled "Method and System for Recognizing Sound and Music Signals in 
High Noise and Distortion" filed on July 31, 2000 also by some of the same inventors. 
Each of these patent applications is hereby incorporated by reference as if repeated herein 
in its entirety, including the drawings. 

COPYRIGHT NOTICE 

[0003] A portion of the disclosure of this patent document contains material that may 
be subject to copyright protection. The copyright owner has no objection to the facsimile 
reproduction of the patent document or the patent disclosure, as it appears in the U.S. 
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Patent & Trademark Office patent file or records, but otherwise reserves all copyright 
rights whatsoever. 

BACKGROUND OF THE INVENTION 

[0004] The present invention relates generally to methods and apparatuses for 
automatically identifying media samples, and more particularly to a method and apparatus 
for automatically identifying a media sample based on a database of known media files by 
comparing certain aspects of the media sample to similarly obtained aspects of the known 
media files. 

[0005] The related applications disclosed various methods and apparatuses for 
identifying media samples, and applications for such identification. At the heart of these 
methods and apparatuses is a database of known media files. Creating the database is an 
expensive proposition. Buying a single copy of all known media files and all new ones as 
they are created while effective is probably cost prohibitive. Simply making copies of 
media files while also effective may in certain instances violate copyright laws in some 
countries. Moreover, the uncertainty of whether certain acts do in fact violate copyright 
laws coupled with the fact that copyright laws vary, sometimes significantly, from country 
to country, makes it difficult to invest in and/or implement a system or method that relies 
upon use of unlicensed media. 

[0006] The present invention is therefore directed to the problem of developing a 
method and apparatus for automatically creating a database of known media files at low 
cost and without violating any copyright laws. 
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SUMMARY OF THE INVENTION 

[0007] The present invention solves these and other problems by providing method 
and apparatus for interacting with an on-line community providing access to a large 
number of media files and a database of metadata related to the media files available from 
its users to process segmented portions of each media file to create additional metadata 
that can be subsequently used to create constellations and fingerprints for each media file 
for use in the recognition algorithm to be employed. By creating processed metadata from 
each file that can be then stored for use in a recognition database or further processed, for 
use in the algorithm being actually employed by the recognition process at a later date, the 
present invention probably avoids violating copyright protections, as the processed 
metadata may not be protectable and as no copy of the original file is created, even 
temporarily, in the process of the present invention. Thus, the present invention enables 
an automatic and inexpensive technique for creating the necessary database for use in the 
media recognition systems set forth in the above-mentioned U.S. Patent Applications. 



BRIEF DESCRIPTION OF THE DRAWINGS 

[0008] FIG 1 shows a block diagram of an exemplary embodiment of a system 
according to one aspect of the present invention. 

[0009] FIG 2 shows a block diagram of a second exemplary embodiment of a system 
according to another aspect of the present invention. 

[0010] FIG 3 shows a block diagram of a third exemplary embodiment of a system 
according to another aspect of the present invention. 
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DETAILED DESCRIPTION 

[0011] It is worthy to note that any reference herein to "one embodiment" or "an 
embodiment" means that a particular feature, structure, or characteristic described in 
connection with the embodiment is included in at least one embodiment of the invention. 
The appearances of the phrase "in one embodiment" in various places in the specification 
are not necessarily all referring to the same embodiment. 

[0012] The present invention provides an automatic technique for creating a database 
of media files against which media samples are compared to identify the media samples. 
An exemplary embodiment of the present invention uses an existing system indirectly 
accessed via an existing company. An exemplary service, http://www.CDDB.com 
actually operates in the "background" - people who insert a CD in their PC usually do it 
through third parties, such as MusicMatch or Winamp. This service 
(http://www.cddb.com) is also known as Gracenote, which has a large community of 
users, to harvest fingerprints, i.e., processed metadata. The user interface to Gracenote 
may occur through a third party, such as MusicMatch.com. For the sake of simplicity 
herein, we will leave out the indirect relationship between the user community and 
Gracenote. 

[0013] Currently, Gracenote has over 1,000 licensees in 35 countries, 20 million 
unique users a month, and more than 800,000 albums and 10 million songs in its 
database; hence, Gracenote is a comprehensive and widely accessible platform for 
delivering worldwide music related content and services. Gracenote is a service paid for 
by the third party explained above, by and for music fans to identify the music they play. 
Gracenote's Content Delivery Engine provides the ability to aggregate and deliver rich 
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third-party content that is directly related to music as it is playing. Gracenote's CDKey 
technology verifies possession of a particular CD and uses it as a key to enable web-based 
applications, such as music lockers or music service providers. This CDKey technology 
can even unlock bonus content from any location, including Gracenote-enabled 
applications, such as media players or websites. 

[0014] Several patents describe this service in particular detail, including U.S. Patent 

No. 5,987,525 and U.S. Patent No. 6,154,773, which relate to features including 
synchronizing visual content with playback of a musical recording at a local computer that 
receives the visual content from a remote computer. U.S. Patent No. 6,061,680 relates to 
a method used to find title and track information in a database by calculating approximate 
length information based on the number and length of tracks on a recording. U.S. Patent 
No. 6,161,132 relates to a method of using a first device to control playback of a 
recording at one or more second devices connected to the first device via a network and 
transmitting output data related to the recording between the first and second devices. 
Each of the above four patents is hereby incorporated by reference as if repeated herein in 
their entirety, including the drawings. 

[0015] Users of this service via the above mentioned website are able to obtain 
metadata from the website based on inserting a CD into a CD player, which then uploads 
certain information to the server, in particular the number of tracks and associated track 
lengths. This metadata is then used to identify the particular media and then extract other 
related metadata from the server related to the identified media, such as artist, genre, etc. 
[0016] By using this service, the present invention permits building of the necessary 
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database to supplement content derived from directly ripping parameters from media 
owned by the system. 

[0017] According to an exemplary embodiment for use with Compact Disks (CDs), 
when a user of the CDDB website inserts a CD into a CD player hooked up to a data 
archiving service, constellation or fingerprint parameters are extracted from the tracks on 
the CD and then sent to the data archiving service. The raw parameter data may then be 
further processed, either immediately or at some later time (including after a prolonged 
storage) into landmark/fingerprint pairs for storage in the recognition database. The raw 
parameter data may be optionally stored in case a new revision of the algorithm for 
enhanced recognition is available. The enhanced recognition algorithm could use the 
stored raw parameter data to generate new enhanced fingerprints, whereby access to the 
original source of raw data (i.e., the original CD) is not necessary for minor algorithm 
updates (e.g., linkage updates). 

[0018] In the case of songs or media not already in the database, another aspect of the 
invention provides for this, which aspect also takes care of major algorithm updates as 
well. Rather than extracting the parameters each time and forwarding them, the remote 
client first queries the database to determine if the song already exists in the database. If 
not, then the remote client extracts and forwards the necessary parameters. If the song is 
already in the database, the remote client determines if the remote client is using a version 
of the algorithm that is more up-to-date than the version used on the song previously 
stored in the database. If so, then the remote client performs the parameter extraction. 
[0019] Another aspect of the present invention allows the remote client to update its 
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algorithm to the latest version by either: (1) updating parameters controlling the 
extraction; or (2) downloading a code update embodying the updated parameter 
extraction. This process is accomplished before updating the song. 
[0020] Of course, the raw parameter data being sent over should be associated with 
the track "metadata" information about the song in order for it to be identified. This 
could be accomplished by: (1) extracting identifying parameters that could be used to look 
up the metadata in a metadata database, such that the metadata database could be queried 
immediately or later. If immediately, then: (2a) the retrieved metadata is sent to the 
recognition server, otherwise if later: (2b) the unique ID is sent to the recognition server, 
whereby the database can be subsequently queried. 

[0021] Hence, for example, a small piggyback program could be distributed with the 
CDDB SDK, so that CDDB clients could incorporate the raw parameter extraction 
process, so that raw parameters could be extracted when the CDDB service is used for 
querying. Or, an extractor could be running independently of CDDB, such that it derives 
the parameters for CDDB to use, but does not actually incorporate CDDB code. In the 
latter case, CDDB is queried after the data packet is received by the recognition database. 
[0022] With this method, the database may be incrementally updated such that songs 
with older versions of the algorithm may be updated when a new algorithm is available, 
and the CD is presented by a user, allowing new raw parameters to be extracted. There is 
the issue that the database would have multiple raw parameter formats with algorithms of 
varying ages. This could be handled by segmenting the recognition landmark/fingerprint 
database, such that songs with the same version of parameter are grouped together. Then, 
an incoming song needs to be analyzed using each of the various fingerprinting algorithm 
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versions in use. Corresponding queries into segments of the database are made with the 
proper fingerprint version. As updated parameters for a song trickle in, the song is 
bumped up into its proper version category. The updates may be forced by directly 
extracting parameters from songs that are immediately available, for example if the 
original source material is available in a readily accessible music archive. If the song is 
not available then the song is updated opportunistically as some user with an updated 
extractor presents the song to the extractor. Over time, the majority of the songs attain the 
most recent format. 

[0023] An exemplary embodiment of a process 1 0 according to one aspect is shown 
in FIG 1 for use with audio media, such as compact disks (CDs). The user places a CD in 
his CD player 1 . A software process 3 installed on the user's personal computer 2, to 
which is connected the CD player 1, extracts the constellation or "raw parameters" from 
the CD tracks as the CD plays, or under control of the software process. These raw 
parameters are then transmitted via the Internet 4 (or other computer network) to a server 
5 coupled to a recognition database 6. The raw parameters are stored in the database 6. 
At some point in time, the raw parameters are processed into fingerprint/landmarks (e.g., 
using linkage) using another software process 7 and then stored either back in the 
database 6 or in random access memory (RAM) for use during a subsequent recognition 
process. This process, however, requires manual identification of the CD. 
[0024] Turning to FIG 2, shown therein is a second exemplary embodiment of a 
process 20 according to another aspect of the present invention. This process 20 relies 
upon the identification capabilities of the Gracenote service or other similar service that 
can identify a CD when placed in one's CD player 1. Simultaneous to the uploading of 
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the constellation or raw parameters to the recognition database 6, the track length and 
number of tracks are sent to the Gracenote server 8 for identification by the CDDB 
software 9 (as disclosed in the above mentioned patents), the result of which is forwarded 
to the recognition database 6. The result can either be returned to the user via the same 
path the request arrived and then uploaded at the end of the raw parameter extraction 
process, or forwarded directly to the recognition database 6 via the Internet 4 and server 5. 
[0025] Thus, as before a user places a CD in the CD player 1 . The CD is identified 
via the CDDB software 9. Metadata for that particular CD is provided to the recognition 
server 5 along with the identification for storage in the recognition database 6 in a record 
associated with the received constellation or raw parameters for that particular CD. As 
before, the constellation or "raw parameters" are extracted and transmitted to the 
recognition database 6 via the Internet 4. This processed metadata is then stored, and 
subsequently further processed into fingerprint/landmarks (e.g., using linkage). Finally, 
the subsequently processed metadata is stored for use by the recognition service. This 
process, however, may result in redundant data arriving at the recognition database 6. 
[0026] Turning to FIG 3, shown therein is an exemplary embodiment of another 
process 30 according to another aspect of the present invention. This process 30 performs 
a check prior to extracting the constellation or raw parameters to determine whether the 
recognition database 6 currently holds the latest version (i.e., the correctly processed 
metadata) of the CD in the user's CD player 1 . If so, no further action is taken. If not, the 
data is uploaded in the same manner described previously. 

[0027] First, the user's CD is identified 31 and then a check is performed to see if the 
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recognition database already has the CD with the most up-to-date raw parameter format. 
If the recognition database already contains the most up-to-date raw parameter for the CD 
at issue, then no further action occurs 32. If the recognition database does not have the 
particular CD, then the constellation or "raw parameter" extraction process occurs. The 
resulting constellation or "raw parameters" (i.e., processed metadata) are then transmitted 
to the recognition database 6, where they are placed in storage and subsequently processed 
into fingerprint/landmarks (e.g. using linkage). The resulting further processed metadata 
is stored in RAM for use by the recognition service. 

[0028] The above explanation relates primarily to extracting raw parameter data from 
(physical) CDs inserted in a personal computer. In a similar fashion, one could extract 
raw parameters using the same process-flows as described from scanning a user's hard 
drive or tapping into streaming music. The associated metadata would come from either 
"file embedded info" (e.g., ID3 tags), or could be directly input by the user. 
[0029] For those who wish to submit their music on a "voluntary" basis directly to 
the recognition server, e.g., as the result of an incentive, such as sales commission, etc, 
the above process would simply upload the constellation or raw parameters independently 
of any other service. This would be particularly useful for obscure material, such as dance 
music, deleted items, etc. 

[0030] One possible way to accomplish this is by building an interface to a 
predetermined web site via which fingerprints could be obtained through scanning either a 
CD or digital file. In this case, duplication of fingerprints would be avoided by quickly 
comparing the new fingerprint against the existing database. In addition, such an 
approach would likely not violate copyright laws. 
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[00311 Another possible embodiment includes a CD player/ripper device or program 
with a built-in constellation extractor. One could rip a CD in terms of constellations in 
about 20-30 seconds. This could happen in the background when nobody is paying 
attention. The constellation coefficients could be sent to the recognition server to build 
up the database. As previously noted, this amount of data would be relatively small. Such 
a ripping program could of course send ID3 tags along with the constellation data. 
[0032] Thus far, the discussion has related to the embodiments in which the 
constellation extraction occurs on the user end side. It is also plausible to assume that this 
extraction could be done by other constituencies such as: 

- the record label (with every new release); 

- the artist (when composing a new song); and 

- retail store (when a new album is put on the shelf). 

[0033] Each of the constituencies mentioned above would probably have an incentive 
to add the mediafile to the recognition database (e.g., selling more music). Moreover, the 
recognition service could provide incentives to these constituencies to voluntarily supply 
this information to the recognition service in the way of directed sales, commissions, etc. 
[0034] Although various embodiments are specifically illustrated and described 
herein, it will be appreciated that modifications and variations of the invention are 
covered by the above teachings and within the purview of the appended claims without 
departing from the spirit and intended scope of the invention. For example, while several 
of the embodiments depict the use of specific data formats and protocols, any formats or 
protocols may suffice. Moreover, while some of the embodiments describe specific 
embodiments of computer, clients, servers, etc., other types may be employed by the 
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invention described herein. Furthermore, these examples should not be interpreted to 
limit the modifications and variations of the invention covered by the claims but are 
merely illustrative of possible variations. 
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